Infrared resolution and contrast enhancement with fusion

ABSTRACT

The present disclosure relates to combination of images. A method according to an embodiment comprises: receiving a visual image and an infrared (IR) image of a scene and for a portion of said IR image extracting high spatial frequency content from a corresponding portion of said visual image. The method according to the embodiment further comprises combining said extracted high spatial frequency content from said portion of the visual image with said portion of the IR image, to generate a combined image, wherein the contrast and/or resolution in the portion of the IR image is increased compared to the contrast and/or resolution of said received IR image.

CROSS REFERENCE TO RELATED APPLICATIONS

This patent application is a continuation of U.S. patent applicationSer. No. 14/922,076, filed Oct. 23, 2015. U.S. patent application Ser.No. 14/922,076 is a continuation of U.S. patent application Ser. No.13/437,645, filed Apr. 2, 2012. U.S. patent application Ser. No.13/437,645 is a continuation-in-part of U.S. patent application Ser. No.12/766,739, filed Apr. 23, 2010. U.S. patent application Ser. No.13/437,645 also claims the benefit of U.S. Provisional PatentApplication No. 61/473,207, filed Apr. 8, 2011. U.S. patent applicationSer. No. 13/437,645 is also a continuation-in-part of U.S. patentapplication Ser. No. 13/105,765, filed May 11, 2011, which is acontinuation of PCT Patent Application No. PCT/EP2011/056432, filed Apr.21, 2011, which is a continuation-in-part of U.S. patent applicationSer. No. 12/766,739, filed Apr. 23, 2010. U.S. patent application Ser.No. 13/105,765 is also a continuation-in-part of U.S. patent applicationSer. No. 12/766,739, filed Apr. 23, 2010. PCT Patent Application No.PCT/EP2011/056432 also claims the benefit of U.S. Provisional PatentApplication No. 61/473,207, filed Apr. 8, 2011. All of the above patentapplications are incorporated herein by reference in their entirety.

TECHNICAL FIELD

The present disclosure relates to a method, imaging device, software,and system for improving an infrared (IR) image.

BACKGROUND

Within the area of image processing, an IR image of a scene comprisingone or more objects can be enhanced by combination with imageinformation from a visual image, said combination being known as fusion.A number of technical problems arise when attempting to accomplish suchcombination and enhancement.

Typically, an imaging device in the form of a camera is provided tocapture a visual image and an IR image and to process these images sothat they can be displayed together. The combination is advantageous inidentifying variations in temperature in an object using IR data fromthe IR image while at the same time displaying enough data from thevisual image to simplify orientation and recognition of objects in theresulting image for a user using the imaging device.

Since the capturing of the IR image and the visual image may beperformed by different components of the imaging device, the opticalaxes between the imaging components may be at a distance from each otherand an optical phenomenon known as parallax will arise. To eliminatethis and the error arising from an angle between the optical axes, theimages must be aligned.

When combining an IR image with a visual image, a number of differentmethods are known. The most commonly used are known as threshold fusionand picture-in-picture fusion.

In a method for performing a threshold fusion of images, a visual imageand an IR image of the same scene are captured. In the IR image, atemperature interval is selected and only those pixels of the image thatcorrespond to temperatures inside the selected interval are chosen anddisplayed together with information data from all other pixels. Theresulting combination image shows the visual image except for thoseareas where a temperature inside the selected interval can be detectedand displays data from the IR image in these pixels instead. Forexample, when a wet stain on a wall is to be detected, a thresholdfusion can be used for determining the extent of the moisture by settingthe temperature threshold to an interval around the temperature of theliquid creating the stain. Other parts of the wall will be closer toroom temperature and will show up as visual data on a screen, so thatthe exact position of the stain can be determined. By seeing a textureof the wall, for instance a pattern of a wallpaper, the location of thestain can be further determined in a very precise way.

When performing picture-in-picture fusion, a visual image and an IRimage showing the same scene comprising one or more objects arecaptured, and the pixels inside a predetermined area, often in the formof a square, are displayed from the IR image while the rest of thecombined image is shown as visual data. For example, when detecting adeviation in a row of objects that are supposed to have roughly the sametemperature, a square can be created around a number of objects andmoved until a faulty object is captured besides a correctly functioningone and the difference will be easily spotted. By displaying elementsfrom the visual image outside this square, such as text or pattern, forinstance, the precise location of the objects with a specifictemperature can be more easily and reliably determined.

The methods for threshold fusion and picture-in-picture fusion alldisplay the chosen section of the combined image as IR data while therest is shown as visual data. This has the disadvantage that detailsthat are visible in the visual image are lost when showing IR data forthe same area. Likewise, temperature data from the IR image cannot beshown together with the shape and texture given by the visual image ofthe same area.

Some methods exist for blending IR data and visual data in the sameimage. However, the results are generally difficult to interpret and canbe confusing to a user since temperature data from the IR image,displayed as different colors from a palette or different grey scalelevels, are blended with color data of the visual image. As a result,the difference between a red object and a hot object, for instance, or ablue object and a cold object, can be impossible to discern. Generally,the radiometric or other IR related aspects of the image, i.e. thesignificance of the colors from the palette or grey scale levels, arelost when blending the IR image with the visual image.

Thus, there exists a need for an improved way of providing a combinedimage comprising data from an IR image and data from a visual imagetogether.

SUMMARY

One or more embodiments of the present disclosure may solve or at leastminimise the problems mentioned above. This is achieved by a method, animaging device, and/or a non-transitory computer program productaccording to the claims, where an IR image is combined with high spatialfrequency content of a visual image to yield a combined image. Accordingto embodiments, the imaging device comprises a processing unit (e.g., aprocessor, a programmable logic device, or other type of logic device)configured to perform any or all of the method steps of the methodembodiments described herein. The combination is performed throughsuperimposition of the high spatial frequency content of the visualimage and the IR image, or alternatively superimposing the IR image onthe high spatial frequency content of the visual image. As a result,contrasts from the visual image can be inserted into an IR image showingtemperature variations, thereby combining the advantages of the twoimage types without losing clarity and interpretability of the resultingcombined image.

More specific aspects of the embodiments of the present disclosure areexplained below.

The method according to an embodiment of the present invention comprisesensuring that the resolutions of the images to be combined, i.e. theresolution of a visual image and an IR image, are substantially thesame. According to embodiments described herein, the images may havesubstantially the same resolution when they are captured, or the imagesmay require processing in order to ensure that they have substantiallythe same resolution before remaining method steps are performed.Embodiments for ensuring that the images have substantially the sameresolution are presented herein.

In a first exemplary embodiment, this (e.g., ensuring that theresolution of a visual image and an a image are substantially the same)may be performed by configuring an imaging device with an IR sensor anda visual image sensor, such that the IR sensor and the visual imagesensor have substantially the same resolution. In another alternativeembodiment, the resolutions of the imaging sensors are previously knownnot to be substantially the same, for example to differ more than apredetermined difference threshold value. In yet another alternativeembodiment, if the resolutions of the imaging sensors are not previouslyknown, the inventive method may include a step of checking whether theresolutions of the received images are substantially the same. Checkingmay be performed through comparison of the resolutions of the receivedimages, wherein information on the resolutions of the images may eitherbe available from the separate imaging sensors or retrievedfrom/calculated based on the received images. If the resolutions arefound not to be substantially the same, either through previousknowledge of the sensor resolutions or through checking of theresolutions, the ensuring that the resolutions are substantially thesame further includes re-sampling of at least one of the receivedimages.

The method according to another embodiment comprises receiving a visualimage and an infrared (IR) image of a scene and for a portion of said IRimage extracting high spatial frequency content from a correspondingportion of said visual image, i.e. corresponding to the portion of theIR image. According to an embodiment, the corresponding portion of thevisual image is the portion that shows the same part of the observedreal world scene as the portion of the IR image. The method embodimentfurther comprises combining said extracted high spatial frequencycontent from said portion of said visual image with said portion of theIR image, to generate a combined image, wherein the contrast and/orresolution in the portion of the IR image is increased compared to thecontrast of said captured IR image.

According to an embodiment, the resolution of the captured visual imageand the resolution of the captured IR image are substantially the same.

According to different embodiments, said portion of the IR image may bethe entire IR image or a sub portion of the entire IR image and saidcorresponding portion of the visual image may the entire visual image ora sub portion of the entire visual image.

According to an embodiment, said portion is predetermined. According toan embodiment, the method further comprises receiving a control signalindicating a manual selection of a portion of said IR image. Accordingto an embodiment, said portion of the IR image is a predetermined areain the IR image.

According to an embodiment, said portion of the IR image and saidcorresponding portion of said visual image are scaled to a predeterminedsize. According to embodiments, said predetermined size is a selectionof: the size of the captured IR image; the size of the captured visualimage; and the size of a display onto which the combined image is to bedisplayed.

According to an embodiment, said portion of the IR image and saidcorresponding portion of said visual image are resampled to match apredetermined resolution. According to embodiments, said predeterminedresolution is a selection of: the resolution of the captured IR image;the resolution of the captured visual image; and the resolution of adisplay onto which the combined image is to be displayed.

Since the resolution of an IR image is generally much lower than that ofa visual image, due to properties of an IR imaging device compared to avisual imaging device, the resolution of the IR image may be up-sampledto be substantially the same as the resolution of the visual image. As aresult, an increased level of detail can be achieved and a more easilyanalysed combined image presented to a user. In another example, thevisual image can be down-sampled to be substantially the same as theresolution of the IR image.

In a further example, both images can be sampled to fit a thirdresolution, if suitable. Both images may originally have substantiallythe same resolution, or the resolution of the images may differ. Afterthe resampling to fit the third resolution however, both images willhave substantially the same resolution. This enables the images to becombined in a manner that is convenient and suitable regardless of howthey are to be displayed. In one example, the third resolution can bethat of a display screen where the combined image is to be displayed.

Additionally, extraction of high spatial frequency content in the visualimage and de-noising and/or blurring of the IR image, or a portion ofthe IR image, may preferably be performed. Typically, this is achievedby high pass filtering the visual image and low pass filtering the IRimage, or the portion of the IR image, by use of spatial filters thatare moved across the images, pixel by pixel. It is evident to a personskilled in the art that other well-known image processing methods may beused to render the same result. As a result of the filtering performedon the IR image, or the portion of the IR image, the IR image, or theportion of the IR image, can be rendered smooth and/or contain a reducedamount of noise compared to the original IR image. Additionally, thehigh spatial frequency content extracted from the visual image containsinformation on large contrasts in the visual image, i.e. information onwhere sharp edges such as object contours are located in the visualimage. The step of performing filtering of the IR image is optional. Themethod for an embodiment of the present invention gives beneficialeffects on the resulting image shown to the user even without thefiltering of the IR image and a user would be able to clearly discernone or more objects in the scene depicted in the IR image, or theportion of the IR image, and the temperature information in connectionwith the imaged scene. However, since sharp edges and noise visible inthe original IR image, or the portion of the IR image, are removed or atleast diminished in the filtering process, the visibility in theresulting image may be further improved through the filtering of the IRimage and the risk of double edges showing up in a combined image wherethe IR image and the visual image are not aligned is reduced.

Besides high pass filtering, examples of methods for extracting highspatial frequency content in an image may include extracting thedifference (commonly referred to as a difference image) between twoimages depicting the same scene, where a first image is captured at onetime instance and a second image is captured at a second time instance,preferably close in time to the first time instance. The two images maytypically be two consecutive image frames in an image frame sequence.High spatial frequency content, representing edges and contours of theobjects in the scene, will appear in the difference image unless theimaged scene is perfectly unchanged from the first time instance to thesecond, and the imaging sensor has been kept perfectly still. The scenemay for example have changed from one frame to the next due to changesin light in the imaged scene or movements of depicted objects. Also, inalmost every case the imaging sensor will not have been kept perfectlystill.

If the imaging device is handheld, it is evident that there will bemovements caused by the user of the imaging device. If the camera isstationary, for example on a stand, vibrations of the imaging device orthe surroundings may cause movements of the imaging sensor. Vibrationsof the imaging device may for example be caused by image stabilizationsystems, which are commonly used in visual imaging devices in order tocompensate for movements of the imaging device. Different ways ofaccomplishing image stabilization is well known in the art. In animaging device having an image stabilization system, the imaging sensormay be placed on an element that enables moving the imaging sensor inresponse to measured movements of the imaging device. This constructioncould be used to capture edges/contours in difference images, if themovements of the imaging sensor are controlled to correspond to acertain, predefined difference between consecutive image frames. In thiscase, the difference may further correspond to a certain width of theedges/contours of the difference image, the width being chosen accordingto circumstances.

Another way of obtaining images from which a difference image can bederived is to use the focus motor of the imaging device to move one ormore lenses of the imaging device. The use of a focus motor for movinglenses in an imaging device is well known in the art. In this case, animage captured by the imaging device when it is slightly out of focuswould be a smoothed and de-noised image that could directly correspondto a low-pass filtered image. After the focus of the imaging device hasbeen reset, a focused image may be captured and the high spatialfrequency content of the focused image may be obtained by subtractingthe out-of-focus image from the focused image.

The approaches of using vibrations of the imaging sensor of refocusingof one or more lenses in the imaging device further do not necessarilyrequire any digital image processing and could therefore be used inconnection with analog imaging devices. As is evident to a personskilled in the art, by subtracting the extracted high spatial frequencycontent obtained by any of the methods described above from an image acorresponding low-pass filtered version of the image is obtained, sinceonly the lower spatial frequency content remains after the subtraction.When combining the images, adding the high pass filtered or extractedhigh spatial frequency content of the visual image, or the portion ofthe visual image, to the IR image, or to the portion of the IR image,adds contours and contrasts to the IR image, or to the portion of the IRimage, but does not otherwise alter it. As a result, the borders andedges of objects captured by the images can clearly be seen in thecombined image, while at the same time maintaining a high level ofradiometry or other relevant IR information.

In one example, to preserve the color or grey scale palette of the IRimage, only the luminance component of the filtered visual image, or theportion of the filtered visual image, may be added to the IR image, orto the portion of the IR image. As a result, the colors are not alteredand the properties of the original IR palette maintained, while at thesame time adding the desired contrasts. To maintain the IR palettethrough all stages of processing and display is beneficial, since theradiometry or other relevant IR information may be kept throughout theprocess and the interpretation of the combined image may thereby befacilitated for the user.

When combining the luminance of the visual image, or the portion of thevisual image, with the IR image, or with the portion of the IR image, afactor alpha can be used to determine the balance between the twoimages. This factor can be decided by the imaging device or imagingsystem itself, using suitable parameters for determining the level ofcontour needed from the visual image to create a good image, but canalso be decided by a user by giving an input to the imaging device orimaging system. The factor can also be altered at a later stage, such aswhen images are stored in the system or in a PC or the like and can beadjusted to suit any demands from the user.

Before displaying the resulting combined image to a user, highresolution noise may be added to the image in order to create animpression of high resolution and increased detail and make the imagemore easily interpreted by the user.

The scope of the invention is defined by the claims, which areincorporated into this Summary by reference. A more completeunderstanding of embodiments of the invention will be afforded to thoseskilled in the art, as well as a realization of additional advantagesthereof, by a consideration of the following detailed description of oneor more embodiments. Reference will be made to the appended sheets ofdrawings that will first be described briefly.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows a flow chart of a method according to an exemplaryembodiment.

FIG. 2 shows a schematic view of a method with the images of thedifferent stages of the method according to an exemplary embodiment.

FIG. 3a shows an IR image in halftone.

FIG. 3b shows the IR image of FIG. 3a in halftone after low passfiltering.

FIG. 3c shows extracted high spatial frequency content of a visual imagein halftone, in this example obtained by high pass filtering.

FIG. 3d shows a combination of the low pass filtered IR image of FIG. 3bwith the high pass filtered visual image of FIG. 3c in halftone.

FIG. 4 shows an exemplary embodiment of an image processing system forperforming a method according to an exemplary embodiment.

FIG. 5a shows the IR image of FIG. 3a with areas of differenttemperatures marked by different patterns.

FIG. 5b shows the image of FIG. 3b with areas of different temperaturesmarked by different patterns.

FIG. 5c shows the image of FIG. 3 c.

FIG. 5d shows the image of FIG. 3d with areas of different temperaturesmarked by different patterns.

FIG. 6a shows a low resolution IR image of a scene. The shown image hasa resolution of 32×32 pixels.

FIG. 6b shows the IR image of FIG. 6a after the IR image has beenre-sampled, processed and combined with extracted high spatial frequencycontent of a visual image depicting the same scene.

FIG. 7a shows a combined image according to an embodiment.

FIG. 7b shows scaling of a portion of an IR image and a resultingcombined image according to an embodiment.

FIG. 8a shows an IR image according to an embodiment.

FIG. 8b shows a combined image according to an embodiment.

Embodiments of the invention and their advantages are best understood byreferring to the detailed description that follows. It should beappreciated that like reference numerals are used to identify likeelements illustrated in one or more of the figures.

DETAILED DESCRIPTION

In FIG. 1, an exemplary method according to an embodiment of the presentdisclosure can be seen. At block 101 a visual image is captured and atblock 102 an IR image is captured. The visual image and IR image may becaptured by an optical sensor and an IR sensor, respectively. Aftercapture, the visual image and the IR image may be aligned at block 103to compensate for the parallax between the optical axes that generallyarises due to differences in placement of the sensors for capturing saidimages and the angle created between these axes because of mechanicaltolerances that generally prevents them being mounted exactly parallel.

The blocks 101, 102 can be performed simultaneously or one after theother. In one example, the images may be captured at the same time orwith as little time difference as possible, since this will decrease therisk for alignment differences due to movements of an imaging deviceunit capturing the visual and IR images. As is readily apparent to aperson skilled in the art, images captured at time instances furtherapart may also be used.

After alignment at block 103, ensuring that the visual image resolutionand the IR image resolution are substantially the same is performed atblock 110. In a first exemplary embodiment, this may be performed byconfiguring an imaging device with an IR sensor and a visual imagesensor, such that the IR sensor and the visual image sensor havesubstantially the same resolution. In another exemplary embodiment, theresolutions of the imaging sensors are previously known not to besubstantially the same, for example to differ more than a predetermineddifference threshold value. In yet another exemplary embodiment, if theresolutions of the imaging sensors are not previously known, theinventive method may include a step of checking whether the resolutionsof the received images are substantially the same at block 109. Checkingmay be performed through comparison of the resolutions of the receivedimages, wherein information on the resolutions of the images may eitherbe available from the separate imaging sensors or retrieved from orcalculated based on the received images. If the resolutions are foundnot to be substantially the same, either through previous knowledge ofthe sensor resolutions or through checking of the resolutions, theensuring that the resolutions are substantially the same furtherincludes re-sampling of at least one of the received images at block104.

According to an embodiment, the method steps are performed for a portionof the IR image and a corresponding portion of the visual image.According to an embodiment, the corresponding portion of the visualimage is the portion that shows the same part of the observed real worldscene as the portion of the IR image. According to differentembodiments, the corresponding part of the visual image may be detectedor identified using alignment and/or stabilization of the images; objector feature detection; and/or known relationships between the imagingdevices 11, 12 (FIG. 4 and described further herein), such as parallaxand pointing errors known from design, production, and/or calibration ofthe imaging device unit 1 (shown and described in reference to FIG. 4).According to an embodiment, the method comprises receiving a visualimage and an infrared (IR) image of a scene and for a portion of said IRimage extracting high spatial frequency content from a correspondingportion of said visual image, i.e. corresponding to the portion of theIR image. In this embodiment, high spatial frequency content isextracted from the portion of the visual image, and the portion of theIR image is combined with the extracted high spatial frequency contentof the portion of the visual image, to generate a combined image,wherein the contrast and/or resolution in the portion of the IR image isincreased compared to the contrast of the originally captured IR image.

According to an embodiment, the resolution of the captured visual imageand the resolution of the captured IR image are substantially the same.

According to different embodiments, said portion of the IR image may bethe entire IR image or a sub portion of the entire IR image and saidcorresponding portion of the visual image may be the entire visual imageor a sub portion of the entire visual image. In other words, accordingto an embodiment the portions are the entire IR image and acorresponding portion of the visual image that may be the entire visualimage or a subpart of the visual image if the respective IR and visualimaging systems have different fields of view.

According to an embodiment, the identified portion of the IR image ispredetermined, e.g. comprising a predetermined area or region of the IRimage. According to another embodiment, the method further comprisesreceiving a control signal indicating a manual selection of a portion ofsaid IR image.

According to another embodiment, the portions comprise a subpart of theIR image and a corresponding subpart of the visual image, respectively,whereby the high frequency content of said subpart of the visual imageis in the last step combined with said subpart of the IR image.According to an embodiment, wherein the portion of the IR image is asubpart of the captured IR image, the resulting combined image can beseen as a “picture in picture” image, comprising the captured IR imageand the IR image portion with increased contrast from adding highfrequency content added from the corresponding portion of the visualimage. An example of a resulting “picture in picture” image according toone or more embodiments of the invention is shown in FIG. 7a , showingan IR image 700 wherein the portion 720, having increased contrastand/or resolution, is indicated with a dotted outline 710, for improvedvisibility. Such an outline or other indicating information mayoptionally be present in the combined image displayed to the user.

According to one or more embodiments, wherein the portions are subpartsof the IR and visual images respectively, the portion of the IR imageand said corresponding portion of said visual image may be scaled to apredetermined size. According to an embodiment, said portion of the IRimage and said corresponding portion of said visual image are resampledto match a predetermined resolution. According to one or moreembodiments, said predetermined resolution is a selection of: theresolution of the captured IR image; the resolution of the capturedvisual image; and the resolution of a display onto which the combinedimage is to be displayed.

According to one or more embodiments, said predetermined size is aselection of: the size of the captured IR image; the size of thecaptured visual image; and the size of a display onto which the combinedimage is to be displayed. In other words, the corresponding portions mayfor instance be scaled to fit the resolution of the captured IR image,the captured visual image, or a display onto which the combined image isto be displayed. According to an embodiment, the portions are scaled tomatch the resolution of the captured IR image. The scaling according todifferent embodiments may be performed either before the extraction ofhigh spatial frequency content described below, directly after theextraction, or after the combination of the high spatial frequencycontent with the portion of the IR image. According to all these scalingembodiments, the scaled version of the combined image may then be storedand or presented to a user on a display integrated in or coupled to theimaging device unit used for capturing the images. In FIG. 7b , anexample of a manual or an automatic/predetermined selection of a portioncorresponding to an area (e.g., portion 720 having dotted outline 710)of the IR image 700 is shown. Furthermore, FIG. 7b shows a resultingcombined image portion 720 that has been scaled to fit a largerresolution, for example the resolution of a certain display. In FIGS. 7aand 7b the portions are shown as rectangular areas. As is readilyapparent to a person skilled in the art, the portions may have anyshape, size and location in the IR image.

In FIGS. 7a and 7b , a low resolution IR image 700 of 32×32 pixels isshown. However, preferably the IR image from which a portion is selectedor identified has a higher resolution, so that the selected oridentified portion will have a resolution of at least 32×32 pixels. Anexample of a higher resolution IR image 800, showing a parking lotfilled with cars, is shown in FIG. 8a . The resolution of the IR image800 is indicated on its x and y axis, respectively. In image 800, twohigh intensity spots seemingly indicating hot spots on the wall of thehouse on the other side of the parking lot are marked by a circle 810.The circle is only there for visibility reasons. In a use caseembodiment, the user may have identified the spots as interesting toinvestigate further. Therefore, the user selects an area or a portion820 to zoom into and perform contrast/resolution enhancement onaccording to any of the method embodiments presented herein. In FIG. 8b, the resulting zoomed-in combined image 830 is shown. On the x and yaxes of the combined image 830 it is indicated which part of the IRimage 800 the selected portion represents. From the combined image 830,the user can easily see that the hotspots that were presumably locatedon the wall of the house are in fact street lights on the parking lot.Thereby, the user's understanding of the content shown in the IR imageis enhanced through the presented method. In FIG. 8b , the combinedimage is shown in a zoomed-in form to the user, e.g. filling the entiredisplay onto which the image is presented. Alternatively, the combinedimage may be presented as a picture in picture image, in the mannershown in FIG. 7 a.

According to an embodiment, the combined image is scaled to completelyfill the display. According to another embodiment, the combined image iswindowed and scaled so that it matches the identified portion of thecaptured visual image. The combined image may according to thisembodiment be displayed over the matching area of the visible-lightimage, thereby providing a picture in picture effect.

Standard image processing techniques, such as scaling and/or windowingfor example, are used to fit the combined image into the desired area ofthe display.

According to an embodiment, the portion of the IR image may be selectedby a user giving a selection input to the imaging device or imagingsystem for example using an input device integrated in or coupled to theimaging device unit (e.g., such as integrated in or coupled to controlunit 42 and/or display 3 of FIG. 4 discussed further herein). The inputdevice may be a selection of buttons, keyboard, soft buttons, a computermouse, touch functionality, a joystick or any other input functionalitythat enables a user to perform a selection in interaction with a userinterface wherein the IR image, the visual image or a combined versionof the two, e.g. a blended or fused image, is shown. According to thisembodiment, the method further comprises receiving a control signalindicating a manual selection of a portion of said IR image, generatedby the input provided by the user.

According to another embodiment, said portion (e.g., portion 720) ispredetermined or set to a default subpart of the captured IR image, forinstance during design, production, and/or calibration of the imagingdevice unit. The predetermined portion may for instance be a subpartlocated in the center of the captured IR image, a subpart of the imagewherein the outermost parts, e.g. a “frame” are not included, or in anyother suitable area selected during setting of the predeterminedportion. According to an embodiment, the predetermined portion may beindicated, marked, and/or highlighted in a graphical user interfaceintegrated in or coupled to a display (e.g., such as integrated in orcoupled to display 3 of FIG. 4 discussed further herein) showing thecaptured IR image or a blended, fused or picture in picture version ofthe captured IR and visual images. For instance, the predeterminedportion may be marked by a frame or outline.

Herein, the term IR image may refer to the originally captured IR image,a portion of the originally captured IR image, or a scaled version ofthe portion of the IR image.

In one exemplary embodiment, the IR image may be re-sampled to increaseor decrease its resolution. Typically, the resolution of a captured IRimage has a different resolution than that of a captured visual image;usually the IR image has a lower resolution than the resolution of thevisual image. A normal resolution for an IR image can for instance be320×240 pixels, while a normal resolution for a visual image can bearound 5 M pixels. If the resolutions of images to be combined are notsubstantially the same, at least one of them may have its resolutionaltered to match the other in order to compensate for the difference andmore successfully combine the images. In one example, this may be doneby up-sampling the IR image to the resolution of the visual imagethrough interpolation. It is also possible to configure an imagingdevice with an IR sensor and a visual image sensor having substantiallythe same resolutions.

As an alternative to up-sampling the IR image, the visual image may bedown-sampled to fit the resolution of the IR image, or both images canbe sampled to fit a third resolution. Both images may originally havesubstantially the same resolution, or the resolution of the images maydiffer. After the resampling to fit the third resolution however, bothimages will have substantially the same resolution. This enables theimages to be combined in a manner that is convenient and suitableregardless of how they are to be displayed.

If the combined image is to be stored and displayed by an IR camera, aPC or other device with a high resolution in for example image datastructures and/or image display means, it can be convenient to up-samplethe IR image to fit the generally higher resolution of the visual image.However, if the combined image is to be displayed by a system with muchlower resolution, it may be more suitable to down-sample the visualimage to fit this requirement. According to an exemplary embodiment, athird resolution may be selected to be the resolution of a displayscreen where the combined image is to be presented. Both images mayoriginally have substantially the same resolution, or the resolution ofthe images may differ. It is, however, beneficial if the resolutions ofthe visual image and the IR image, respectively, are substantially thesame before the images are to be combined, so that a suitable matchingof data for each pixel of the images can be performed.

At block 105, the high spatial frequency content of the visual image maybe extracted, for example by high pass filtering the visual image usinga spatial filter. Besides high pass filtering, examples of methods forextracting high spatial frequency content in an image may includeextracting the difference (commonly referred to as a difference image)between two images depicting the same scene, where a first image iscaptured at one time instance and a second image is captured at a secondtime instance, preferably close in time to the first time instance. Thetwo images may typically be two consecutive image frames in an imageframe sequence. High spatial frequency content, representing edges andcontours of the objects in the scene, will appear in the differenceimage unless the imaged scene is perfectly unchanged from the first timeinstance to the second, and the imaging sensor has been kept perfectlystill. The scene may for example have changed from one frame to the nextdue to changes in light in the imaged scene or movements of depictedobjects. Also, in almost every case the imaging sensor will not havebeen kept perfectly still.

If the imaging device is handheld, it is evident that there will bemovements caused by the user of the imaging device. If the camera isstationary, for example on a stand, vibrations of the imaging device orthe surroundings may cause movements of the imaging sensor. Vibrationsof the imaging device may for example be caused by image stabilizationsystems, which are commonly used in visual imaging devices in order tocompensate for movements of the imaging device. Different ways ofaccomplishing image stabilization is well known in the art. In animaging device having an image stabilization system, the imaging sensormay be placed on an element that enables moving the imaging sensor inresponse to measured movements of the imaging device. This constructioncould be used to capture edges/contours in difference images, if themovements of the imaging sensor are controlled to correspond to acertain, predefined difference between consecutive image frames. In thiscase, the difference may further correspond to a certain width of theedges/contours of the difference image, the width being chosen accordingto circumstances.

Another way of obtaining images from which a difference image can bederived is to use the focus motor of the imaging device to move one ormore lenses of the imaging device. The use of a focus motor for movinglenses in an imaging device is well known in the art. In this case, animage captured by the imaging device when it is slightly out of focuswould be a smoothed and de-noised image that could directly correspondto a low-pass filtered image. After the focus of the imaging device hasbeen reset, a focused image may be captured and the high spatialfrequency content of the focused image may be obtained by subtractingthe out-of-focus image from the focused image.

The approaches of using vibrations of the imaging sensor of refocusingof one or more lenses in the imaging device further do not necessarilyrequire any digital image processing and could therefore be used inconnection with analog imaging devices. As is evident to a personskilled in the art, by subtracting the extracted high spatial frequencycontent obtained by any of the methods described above from an image acorresponding low-pass filtered version of the image is obtained, sinceonly the lower spatial frequency content remains after the subtraction.

At block 106, the IR image may be processed in order to reduce noise inthe image and/or blur the image, for example through the use of aspatial low pass filter. Low pass filtering may be performed by placinga spatial core over each pixel of the image and calculating a new valuefor said pixel by using values in adjacent pixels and coefficients ofsaid spatial core. In another example, the images may be filtered usingsoftware alone.

A spatial low pass filter core can be a 3×3 filter core with thecoefficient 1 in every position, and the filtered value of a pixel canbe calculated by multiplying an original pixel value and eight adjacentpixels each by their filter coefficient, adding them together, anddividing by 9. After performing this operation for each pixel in an IRimage, a low pass filtered image with a smoother appearance can becreated. For high pass filtering an IR image, the same filtercoefficients can be used, such that the high pass filtered image isformed by subtracting the low pass filtered image from the originalimage, one pixel at a time, in a manner well-known in the art. It is tobe noted, however, that the coefficients of the filter core can be setto different values, and that a size of the filter core can be otherthan the 3×3 filter core described above. The resulting processed visualimage and the possibly processed IR image may be combined at block 107.Before displaying the resulting combined image high resolution noise,for example high resolution temporal noise, may be added at block 108.

The step of performing filtering of the IR image at block 106 isoptional. The method for an embodiment of the present invention givesbeneficial effects on the resulting image shown to the user even withoutthe filtering of the IR image and a user would be able to clearlydiscern objects in the imaged scene as well as temperature informationof the imaged scene. The purpose of the low pass filtering performed atblock 106 is to smooth out unevenness in the IR image from noise presentin the original IR image captured at block 102. Since sharp edges andnoise visible in the original IR image are removed or at leastdiminished in the filtering process, the visibility in the resultingimage is further improved through the filtering of the IR image and therisk of double edges showing up in a combined image where the IR imageand the visual image are not aligned is reduced.

A high pass filtering is performed for the purpose of extracting highspatial frequency content in the image, in other words locating contrastareas, i.e. areas where values of adjacent pixels display largedifferences, such as sharp edges. A resulting high pass filtered imagecan be achieved by subtracting a low pass filtered image from theoriginal image, calculated pixel by pixel, as will also be described indetail below.

As is readily apparent to a person skilled in the art, after the methodsteps according to any of the embodiments presented herein have beenperformed, the resulting image or parts of the resulting image may befurther processed according to methods per se known in the art. FIG. 2shows an exemplary embodiment of images that are produced at differentblocks of the method illustrated by FIG. 1. A visual image 301 that iscaptured at block 101 and an IR image 302 captured at block 102 are usedas input for up-sampling and filtering during processing 303,corresponding to blocks 103,104, 105, 106.

After processing 303, extracted high spatial frequency content 304 ofthe visual image is shown, where the contours of objects present in theoriginal visual image 301 can be seen. According to an exemplaryembodiment of the present invention, the IR image 302 is processed intoa low pass filtered and up-sampled image 305. The up-sampling hasincreased the resolution of the image and now each object in the imagedscene can be seen more clearly, without showing much noise in the formof blurs or graininess in the low pass filtered image 305. Arrows fromthe extracted high spatial frequency content of the visual image 304 andthe low pass filtered IR image 305 that can now be described asprocessed images 304, 305, indicate a combination of these images 304,305 to form a combined image 307 where the processed IR image 305,displaying the smooth temperature distribution in the imaged scene iscombined with the processed visual image 304 where the contours or edgesfrom objects of the original visual image 301 are also shown. Thecombined image 307 thus displays the advantages of the IR image 302,where any differences in temperature across the objects are shown, withthe contours from the processed visual image 304 in order to show theshape of each object more clearly. The combination is preferablyperformed through either superimposing the high spatial frequencycontent of the visual image on the IR image, or alternativelysuperimposing the IR image on the high spatial frequency content of thevisual image.

According to an exemplary embodiment of the present invention, the IRimage may be captured with a very low resolution IR imaging device, theresolution for instance being as low as 64×64 or 32×32 pixels, but manyother resolutions are equally applicable, as is readably understood by aperson skilled in the art.

According to another embodiment, a portion of the IR image having a sizeof 64×64 pixels or less, or even 32×32 pixels or less, is identified inthe IR image. According to an embodiment, the location/area in the IRimage representing the portion may be predetermined or determined basedon manual input.

A 32×32 pixel IR image in itself contains very little information and itis hard for a viewer to interpret the information in the image. Anexample of an IR image having the resolution of 32×32 pixels is shown inFIG. 6a . The inventor has found that if edge and contour (high spatialfrequency) information is added to the combined image from the visualimage, the use of a very low resolution IR image will still render acombined image where the user can clearly distinguish the depictedobjects and the temperature or other IR information related to them.FIG. 6b shows the IR image of FIG. 6a after the IR image has beenre-sampled and combined with extracted high spatial frequency content ofa visual image depicting the same scene. This enables the inventivemethod for an embodiment to be used in combination with very small andinexpensive image detectors, still rendering very advantageous results.

According to another exemplary embodiment, the IR image may be capturedwith a high resolution IR imaging device. As the technology advances, IRimaging devices continue to get higher resolution. A high resolution IRimaging device today would for instance have a resolution of 640×640pixels. An IR image captured with such a high resolution imaging devicemay possibly in itself be sufficient to show edge and contourinformation to a viewer. By combining such a high resolution IR imagewith the high spatial frequency content of a corresponding visual imagemay enable the viewer to see further details of the visual image, notshown in the IR image. For example, an area where water damage has beenidentified may be drawn/outlined on a wall using a pen. This informationmay be advantageous to have in combination with the measured temperatureinformation. In another example, there may be a serial number or otheridentifying letters or digits in the image that may help in identifyingthe depicted scene or objects in the scene. A high resolution IR imagemay further advantageously be down-sampled and/or de-noised/low passfiltered to a high degree, whereby the resulting processed IR imagewould contain a very low level of noise, but still has very highsensitivity when it comes to the temperature information of the depictedscene.

High resolution noise 306 may be added to the combined image 307,corresponding to block 108, in order to render the resulting image moreclearly to the viewer and to decrease the impression of smudges or thelike that may be present due to noise in the original IR image 302 thathas been preserved during the low pass filtering of said IR image 302.

FIG. 3a shows an IR image 302 immediately after capture at block 102.The imaged scene represents a bookcase with binders arranged in rows andwith shelves fitted at certain heights. As can be seen, the objects inthe scene are at different temperatures, shown as different sections,where the uppermost parts of the image and the binders placed on themiddle shelf are warmer than the lower shelf or the areas beside andabove the binders. The actual shapes of the objects depicted aredifficult to discern, since no contours of the objects other than thelines between different temperatures are displayed. It would thereforebe very difficult for a user confronted with this image alone toidentify a specific object of a certain temperature. The IR image hasbeen colored according to a chosen color space (described furtherbelow), by adding color to the signal after filtering.

In an exemplary embodiment of the present invention, the captured IRimage is processed through low pass filtering. FIG. 3b shows a low passfiltered IR image 305. The spatial filtering has smoothed out unevennessin the captured IR image 302 and thereby made it easier to differentiatebetween different objects in the scene. Further, the filtering hasremoved noise from the image 302. Also, the edges between these objectshave been smoothed out. This may be done since contours are to be addedfrom the filtered visual image 304, and any alignment error between theimages would otherwise result in double contours that might bedistracting to a viewer.

In an exemplary embodiment of the present invention, the high spatialfrequency content of the captured visual image is extracted by high passfiltering of the visual image. Such a high pass filtered visual image304 that is the result of high pass filtering the captured visual image301, is shown in FIG. 3c . In the high pass filtered visual image 304,mainly the contours and edges of the objects in the scene imaged in theoriginal visual image 301 can be seen. The contours of and edges betweenobjects as well as lines such as text on the binders or patterns fromthe books are visible.

FIG. 3d shows a combined image 307 after the original IR image 302 hasbeen up-sampled, low pass filtered, and combined with a high passfiltered visual image of the same scene. The areas of differenttemperatures can still be seen, but the borders between them have becomeclearer and contour lines for the binders and the shelves have beenadded, originating from the high pass filtered visual image and showingdetails that cannot be seen in an IR image, such as text or other visualpatterns. An increased clarity also comes from the low pass filtering ofthe IR image, where noisy pixels within larger fields of differenttemperature have been smoothed out to form larger areas that are moresimilar. As a result, at least a portion of the noise that may arisefrom the conditions under which the original image was captured can beeliminated.

FIGS. 5a-5d depict the images of FIGS. 3a-3d described above, but in amanner where areas of different temperature are marked by differentpatterns, instead of in halftone. Everything that is said with referenceto FIGS. 3a-3d can thus be directly applied to FIGS. 5a-5d ,respectively.

The low pass filtering that is performed on the IR image 302 may beperformed by using a spatial filter with a suitable filter core, inorder to calculate a new value for each pixel depending on the previousvalue and those of the surrounding pixels. The high pass filtering isgenerally performed by applying a low pass filter and subtracting theresulting low pass filtered image from the original image, leaving onlylines and edges to be seen in the high pass filtered image. Aspreviously mentioned, methods of applying spatial filters are well knownin the art and any such method may be used.

When choosing a palette, for instance according to the YCbCr family ofcolor spaces, the Y component (i.e. the luminance) may be chosen as aconstant over the entire palette. In one example, the Y component may beselected to be 0.5 times the maximum luminance. As a result, whencombining the IR image according to the chosen palette with the visualimage, the Y component of the processed visual image 304 can be added tothe processed IR image 305 and yield the desired contrast without thecolors of the processed IR image 305 being altered. The significance ofa particular nuance of color is thereby maintained during the processingof the original IR image 302.

When calculating the color components, the following equations can beused to determine the components Y, Cr and Cb for the combined image 307with the Y component from the high pass filtered visual image 304 andthe Cr and Cb components from the signal of the IR image 305.hp_y_vis=highpass(y_vis)(y_ir,cr_ir,cb_ir)=colored(lowpass(ir_signal_linear))which in another notation would be written as:hp _(y) _(vis) =highpass(y _(vis))(y _(ir) ,cr _(ir) ,cb _(ir))=colored(lowpass(ir _(signal linear)))

Other color spaces than YCbCr can, of course, also be used withembodiments of the present disclosure. The use of different colorspaces, such as RGB, YCbCr, HSV, CIF, 1931 XYZ or CIELab for instance,as well as transformation between color spaces is well known to a personskilled in the art. For instance, when using the RGB color model, theluminance can be calculated as the mean of all color components, and bytransforming equations calculating a luminance from one color space toanother, a new expression for determining a luminance will be determinedfor each color space.

In one embodiment, block 107 of combining the processed visual image 304with the processed IR image 305 can be performed using only theluminance component Y from the processed visual image 304.

It is to be noted that the blocks of the method described above can beperformed in different order if suitable in accordance with one or moreembodiments.

FIG. 4 shows a schematic view of an embodiment of an image processingsystem for performing a method according to the present disclosure. Animaging device unit 1 may comprise a visual imaging device 11 having avisual sensor and an IR imaging device 12 having an IR sensor that aremounted so that an optical axis of the visual sensor of visual imagingdevice 11 is at a distance d from the IR sensor of IR imaging device 12.The visual imaging device may be any known type of visual imagingdevice, for example a CCD imaging device, an EMCCD imaging device, aCMOS imaging device or an sCMOS imaging device. The IR imaging devicemay be any kind of imaging device that is able to detect electromagneticradiation at least, for example, in the interval between 0.7 and 20 μm.The visual imaging device has a visual field of view α of approximately53°, while the IR imaging device has a visual field of view β ofapproximately 24°. It should be appreciated by one of ordinary skillthat other viewing angles may be used, for example through use ofreplaceable optical elements or lenses including optical elements. ForIR imaging devices, replaceable optical elements or lenses includingoptical elements may for instance render a field of view of 15-45°.

Blocks 101, 102, i.e. the capturing of a visual image 301 and an IRimage 302 may be performed by the imaging device unit 1, and thecaptured images are transmitted to a processing unit 2, also referred toas a processor, where the remaining blocks are performed. According to afurther embodiment, the optional step in block 106 of FIG. 1, i.e.reducing noise and blurring a captured IR image, may be performed byimage processing means or an image processor incorporated in the imagingdevice, where after the visual image and the processed IR image aretransmitted to a processing unit 2, where the method steps of theremaining blocks of FIG. 1 are performed. Said processing unit 2 may bea processor such as a general or special purpose processing engine suchas, for example, a microprocessor, microcontroller or other controllogic or an FPGA unit (Field-programmable gate array) that comprisessections of code, stored on a computer readable storage medium, that arefixed to perform certain tasks but also other sections of code, storedon a computer readable storage medium, that can be altered during use.Such alterable sections can comprise parameters that are to be used asinput for the various tasks, such as the calibration of the IR imagingdevice 12, the alignment for the visual imaging device 11 and IR imagingdevice 12, the sample rate or the filter for the spatial filtering ofthe images, among others.

In this document, the terms “computer program product” and“computer-readable storage medium” may be used generally to refer tonon-transitory media such as memory 41, the storage medium of processingunit 2, or the storage medium of control unit 42. These and other formsof computer-readable storage media may be used to provide instructionsto processing unit 2 for execution. Such instructions, generallyreferred to as “computer program code” or computer program code portions(which may be grouped in the form of computer programs or othergroupings) are adapted to control a data processing system to performany or all of the method steps and functions of the inventive method, asdescribed above. Thus when executed, the computer program code portionsenable the imaging device unit 1 or a computer to perform features orfunctions of embodiments of the current technology. Further, as usedherein, processing logic or logic may include hardware, software,firmware, or a combination of thereof.

The processing unit 2 communicates with a memory 41 where suchparameters are kept ready for use by the processing unit 2, and wherethe images being processed by the processing unit 2 can be stored if theuser desires. Memory 41 may be a random access memory (RAM), a registermemory, a processor cache, a hard disk drive, a floppy disk drive, amagnetic tape drive, an optical disk drive, a CD or DVD drive (R or RW),or other removable or fixed media drive. The memory 41 in turncommunicates with a control unit 42 where said parameters originate, forinstance through input from a calibration file 43 that can be suppliedfrom a manufacturer, by parameters being supplied by the imageprocessing system itself, such as for instance data from a sensor or thelike regarding the distance from the imaging device unit 1 to an objectwhose image is captured, or by parameters being supplied by the user.The control unit 42 can be a programmable unit and determine theparameters needed for performing exemplary methods and how suchparameters should interact with the processing unit 2 and store theseparameters in the memory 41 for easy retrieval by the processing unit 2.

After the processing unit 2 has performed the operation of aligning theimages (block 103), up-sampling the original IR image 302 to generate anup-sampled IR image (block 104), high pass filtering of the originalvisual image 301 to generate a processed visual image 304 (block 105),low pass filtering of the up-sampled IR image to generate a processed IRimage 305 (block 106), combining the processed visual image 304 with theprocessed IR image 305 to generate a combined image 307 (block 107), andadding high frequency noise to this combined image 307 (block 108), theresulting image is presented in a display unit 3 in order to be viewedby the user of the image processing system. If desired, the user cansave the combined image 307 or any of the other images corresponding tothe different method steps to the memory 41 for later viewing or fortransfer to another unit, such as a computer, for further analysis andstorage.

According to embodiments of the present invention, the processing unit 2may be adapted or configured to perform any or all of the method stepsor functions described above.

In an alternative embodiment, disclosed methods can be implemented by acomputing device such as a PC that may encompass the functions of anFPGA-unit specially adapted for performing the steps of the method forone or more embodiments of the present invention, or encompass a generalprocessing unit 2 according to the description in connection with FIG.4. The computing device may further comprise the memory 41 and controlunit 42 and also the display unit 3. It would be possible to use thedisclosed methods live, i.e. for a streamed set of images filtered andcombined in real time, for instance at 30 Hz, that can be recorded andreplayed as a movie, but it would also be possible to use stillpictures.

In one example, the user may be allowed to alter a positive factor alphafor determining how much of the luminance from the visual image 301, 304that is to be used for combining with the IR image 302, 305, forinstance by using the equation below. The luminance Y of the combinedimage 307 is achieved by adding the luminance of the processed IR image305 to the luminance of the highpass filtered visual image multiplied bya factor alpha. The combined components Cr and Cb are taken directlyfrom the IR image 302, 305 and are therefore not affected by thisprocess. If another color space is used, the equations are of coursetransformed before use.comb_y=y_ir+alpha×hp_y_viscomb_cr=cr_ircomb_cb=cb_irwhich in another notation would be written as:comb_(y) =y _(ir)+alpha*hp _(y) _(vis)comb_(cr) =cr _(ir)comb_(cb) =cb _(ir)

The variation of alpha thus gives the user an opportunity to decide howmuch contrast is needed in the combined image. With an alpha of close tozero, the IR image alone will be shown, but with a very high alpha, verysharp contours can be seen in the combined image. Theoretically, alphacan be an infinitely large number, but in practice a limitation willprobably be necessary, to limit the size of alpha that can be chosen towhat will be convenient in the current application.

The up-sampling of the resolution of the IR image 302 at block 104 canalternatively be performed as a down-sampling of the visual image 301 tomatch the resolution of the IR image 302, or indeed a combination of anup-sampling of the IR image 302 and a down-sampling of the visual image301 to a resolution that none of the images 301, 302 originally have, aslong as the result is that the IR image 302 and the visual image 301have the same resolution after the sampling step. It may be convenientto determine the resolution depending on the display area such as thedisplay unit 3 where the combined image 307 is to be displayed and tosample the image or images 301, 302 to match the resolution to the mostsuitable for the display unit 3.

It will be appreciated that, for clarity purposes, the above descriptionhas described embodiments of the technology with reference to differentfunctional units and processors. However, it will be apparent that anysuitable distribution of functionality between different functionalunits, processors or domains may be used without detracting from thetechnology. For example, functionality illustrated to be performed byseparate processors or controllers may be performed by the sameprocessor or controller. Hence, references to specific functional unitsare only to be seen as references to suitable means for providing thedescribed functionality, rather than indicative of a strict logical orphysical structure or organization.

The present disclosure is not to be seen as limited by the embodimentsdescribed above, but can be varied within the scope of the claims, aswill be readily understood by the person skilled in the art.

The invention claimed is:
 1. An imaging system, comprising: an infrared(IR) imaging device configured to capture an IR image representing ascene; a visual imaging device configured to capture a visual image ofat least a portion of the scene, wherein optical axes of the IR imagingdevice and the visual imaging device are located at a distance and anangle from each other; and a processor configured to: align the IR imagewith the visual image at least by compensating for a parallax and/orpointing errors due to the optical axes being located at the distanceand the angle; locate contours and/or edges from the visual image toobtain image data representing the located contours and/or edges; andmodify a luminance component of the IR image based on the image datarepresenting the located contours and/or edges from the visual image toenhance the IR image.
 2. The imaging system of claim 1, wherein: the IRimage and the visual image after the aligning show a same part of thescene; and the processor is configured to determine the parallax and/orpointing errors by performing feature detection and/or based on apre-stored relationship between the IR and visual imaging devices knownfrom design, production, and/or calibration of the IR and visual imagingdevices.
 3. The imaging system of claim 1, wherein: the IR imagerepresents the scene by colors or greyscale levels according totemperature; and the colors or the greyscale levels representing thescene for at least a portion of the IR image are unaltered by themodifying.
 4. The imaging system of claim 1, wherein: the IR imagefurther comprises at least one color component; and temperaturevariations in the scene is represented only by the at least one colorcomponent of the IR image, such that the temperature variations in thescene are shown unaltered in the IR image after the modification by theprocessor.
 5. The imaging system of claim 1, wherein the IR image is agreyscale IR image representing temperature variations in the scene bythe luminance component.
 6. The imaging system of claim 1, wherein: theimage data representing the located contours and/or edges from thevisual image comprises luminance data for the located contours and/oredges; and the processor is configured to modify the luminance componentat least by adding the luminance data for the located contours and/oredges to the luminance component of the IR image.
 7. The imaging systemof claim 1, wherein the contours and/or edges located from the visualimage comprise lines, visual patterns, contours, and/or edges of one ormore objects in the scene.
 8. The imaging system of claim 1, wherein theprocessor is configured to locate the contours and/or edges from thevisual image at least by high-pass filtering the visual image to extracthigh-spatial frequency content comprising the contours and/or edges. 9.A method, comprising: receiving an infrared (IR) image representing ascene, wherein the IR image is captured by an IR imaging device;receiving a visual image of at least a portion of the scene, wherein thevisual image is captured by a visual imaging device; aligning the IRimage with the visual image, wherein the aligning compensates for aparallax and/or pointing errors caused by optical axes of the IR andvisual imaging devices that are located at a distance and an angle fromeach other; locating contours and/or edges from the visual image toobtain image data representing the located contours and/or edges; andmodifying the IR image based on the image data representing the locatedcontours and/or edges from the visual image to enhance the IR image. 10.The method of claim 9, wherein: the IR image and the visual image afterthe aligning show a same part of the scene; and the aligning comprisesdetermining the parallax and/or pointing errors by performing featuredetection and/or based on a pre-stored relationship between the IR andvisual imaging devices known from design, production, and/or calibrationof the IR and visual imaging devices.
 11. The method of claim 9,wherein: the IR image represents the scene by colors or greyscale levelsaccording to temperature; and the colors or the greyscale levelsrepresenting the scene for at least a portion of the IR image areunaltered by the modifying.
 12. The method of claim 11, wherein: the IRimage comprises a luminance component; and the modifying of the IR imagecomprises modifying the luminance component of the IR image based on theimage data representing the located contours and/or edges from thevisual image.
 13. The method of claim 12, wherein: the IR image furthercomprises at least one color component; and temperature variations inthe scene is represented only by the at least one color component of theIR image, such that the temperature variations in the scene are shownunaltered in the IR image after the modifying.
 14. The method of claim12, wherein the IR image is a greyscale IR image representingtemperature variations in the scene by the luminance component.
 15. Themethod of claim 12, wherein: the image data representing the locatedcontours and/or edges from the visual image comprises luminance data forthe located contours and/or edges; and the modifying of the luminancecomponent comprises adding the luminance data for the located contoursand/or edges to the luminance component of the IR image.
 16. The methodof claim 9, wherein the contours and/or edges located from the visualimage comprise lines, visual patterns, contours, and/or edges of one ormore objects in the scene.
 17. The method of claim 9, wherein thelocating of the contours and/or edges comprises high-pass filtering thevisual image to extract high-spatial frequency content comprising thecontours and/or edges.
 18. The imaging system of claim 1, wherein theoptical axes of the IR imaging device and the visual imaging device arenot parallel.
 19. The method of claim 9, wherein the optical axes of theIR imaging device and the visual imaging device are not parallel.